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1. Introduction 



One application of the local stable manifold theorem in optimal control problems is 
that the stable manifold of the associated Hamiltonian dynamics describes the graph 
of the optimal cost. The optimal cost and the optimal control satisfy the Dynamic 
Programming Equations (DPE), which are obtained from the optimal control problem 
of minimizing a discrete-time, nonlinear cost subject to a nonlinear discrete-time 
dynamics through the dynamic programming technique. In proving the existence of 
the solutions of the DPE, we will use the Pontryagin Maximum Principle (PMP) 
which gives the bidirectional nonlinear Hamiltonian dynamics and the condition for 
a control to be optimal satisfied by the optimal state and costate trajectories. 

In the case in which the Hamiltonian matrix of the dynamics is invertible, the 
nonlinear Hamiltonian dynamics can be rewritten as a dynamics with both state and 
costate dynamics propagating in the direction where time approaches infinity. As- 
suming the invertibility of the Hamiltonian matrix is to exclude zero as a closed loop 
eigenvalue. The formal solutions to the DPE have been worked out using Al'brecht's 
method (H) in and are valid for all closed loop eigenvalues lying inside the unit 
circle. To our knowledge the local stable manifold theorem has only been proven 
for invertible maps, see (^Jji El)- Our main result is the extension of the local 
stable manifold theorem to the bidirectional, discrete-time, nondiffeomorphic, non- 
linear Hamiltonian dynamics. This generalizes the proof of the existence of the local 
solutions to the DPE found in pi] . 

For invertible maps, Hartmann )12| had shown the existence of a stable manifold 
by the method of successive approximations on the implicit functional equation. An- 
other method, developed by Kelley is the technique of using the Contraction 
Mapping Theorem on a complete space. There is another method by Irwin |13| based 
on an application of the inverse function theorem on a Banach space of sequences. 
After a two-step process of diagonalizing the bidirectional discrete-time Hamiltonian 
dynamics, we apply the technique of Kelley jl4l on a complete space of Lipschitz 
functions endowed with the supremum norm. 

The paper is organized as follows. In the next section we introduce the bidirectional 
discrete-time Hamiltonian dynamics from the Pontryagin Maximum Principle that is 
associated with the optimal control problem. In Section 3 we give some discussion of 
Gronwall's inequalities in the discrete-time case, which will then be used in the proof. 
In Section 4 we state and prove the local stable manifold theorem. In Section 5 we 
discuss the eigenstructure and the symplectic properties of the dynamics. Finally, 
in Section 6 we show how the Theorem along with these properties give the local 
solvability of the Dynamic Programming Equations. 



We formulate a discrete in time infinite horizon optimal control problem of mini- 
mizing the cost functional. 



2. Nonlinear Dynamics 



oo 




subject to the dynamics 



X 



,+ 



x{0) = xo 
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where the state vector x € 

(2.1) f{x,u) = 

(2.2) l{x,u) = 



r, the control u G M™, and 
Ax + Bu + /[^l {x, u) + /[^l {x,u) + ... 
—x'Qx + x' Su + —uRu + (x, u) + 



where f^"^\x,u) and ?[™l(a;,u) as homogeneous polynomials in x and u of degree m. 
We let a;+ ~ -Tfc+i and x = Xk- 

Associated with optimal control problem formulation there is a nonlinear Hamil- 
tonian, 



(2.3) 



H{x, u, A+) = {X+Yfix, u) + l{x, u) 



where A+ = A^+i and the functions / and g are given by equations H2.1|l and 1)2. 2|l . 
The Pontryagin Maximum Principle (PMP) states the following: 



Theorem 2.1. // Xk and Uk are optimal for fc G 0, 1, 2, . 
for fc G 0, 1, 2, . . . such that 



then there exists Xk 7^ 



(2.4) 

(2.5) 

and 

(2.6) 



A = 



dH 
dH 



{x,u,X+) 



dx 



{x,u,X+) 



argmin^H{x, u, X^ 



Thus, the minimizcr of the nonlinear Hamiltonian evaluated at the optimal x and 
A+ is the optimal control u* amongst all admissible controls u. Note that m*(x, A+) G 
since / G C"^ and I G C in 12.3|l . We assume that H is convex in u to guarantee 
a unique optimal control. With the Hamiltonian H2.3|l . the equations (|2.4|) and (|2.5|) 
are the following 

A-BR-^S' ~BR-^B' 
Q - SR-^S' A' - SR-^B' 



(2.7) 



" x+ ' 




A 







X 




■ F{x,X+) ' 




X+ 


+ 


G{x,X+) 



where x, A G M" and F{x,X^) and G(a;,A+) contain the nonlinear terms. Observe 
the opposing directions of the propagation of the state and costate dynamics in l|2.7|l . 

If we linearize the 2n dimensional difference equations 1)2. 7|l around zero, we obtain 
exactly the linear Hamiltonian system. 



(2.8) 
where 



" x+ ' 




X 


X 


= H 


_ A+ _ 



A - BR-^S' 
Q - SR-^S' 

is the associated Hamiltonian matrix and the corresponding Hamiltonian is 



-BR-^B' 
A' - SR-^B' 



H{x, A^, m) = A^ {Ax + Bu) + —x Qx + x Su + —u Ru. 

In fact, the stable subspace of H is described by 

X^Px 
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where the nonegative definite P satisfies the discrete-time algebraic Riccati equation 
(DTARE), 

(2.9) P - a' PA + {A'pB + S){b'pB + R)-^){X PB + 5)' - Q = 0. 
Moreover, the stable linear subspacc is 




where Eg is spanned by the n stable eigenvalue of the Hamiltonian matrix lying inside 
the unit circle. A detailed proof is found in |24| . Since the linear part of the nonlinear 
bidirectional dynamics is the linear Hamiltonian system, we have that the linear part 
of the local stable manifold is A = Px. 



3. Discrete-Time Version of Gronwall's Inequalities 

We first discuss various results based on the discrete-time version of Gronwall's 
inequalities as these following lemmas will be useful in the proof of the local existence 
of a stable manifold. 

Lemma 3.1. Suppose the sequence of scalars {ujj'j^o satisfies the difference inequal- 
ity 

(3.10) Uk+i < Suk + L 

where (5, L > 0, then 

k-l 
3=0 

The proof of the lemma above clearly follows from summing equation ()3.10|1 from 
1 to k. 

Lemma 3.2. Suppose {£,j}°°^Q is a sequence that satisfies 

k~l 

\^k\<C,J2\i,\ + C2 

3=0 

with constants Ci,C2 > 0. Then 

k 

|6l<C2^(l + Ci)^". 

J = l 

Proof: Let Sk = X]^=o I^jI- Then, the sequence {sj}°^Q satisfies 

Sk+l < (l + Ci)Sfc+C2 

where Ci, C2 > 0. By Lemma [3. 21 

k~l 

\sk\ < (1 + Ci f\so\ + C2 ^(1 + Ci)'=-i-^'. 

3=0 
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It follows that 



161 < (1 + Ci)|s,|+C2 

< (1 + Ci) [(1 + Ci)'^>o| + C2 5^(1 + 



since |so| = 0. 



fc-i 



< C2 5Z(l+Cl)'' 



i=i 



4. Local Stable Manifold Theorem for the Bidirectional 
Discrete-Time Dynamics 

In this section we prove the existence of a local stable manifold 

(4.11) A = 0(.t) 

for the Hamiltonian dynamics, 

A -BR-^B' 
Q A' 



(4.12) 



" x+ ' 




A 







X 




■ ^^(^,A+) " 




X+ 


+ 


_ G(x,A+) _ 



where x, A £ R" and zero is an eigenvalue of A. The nonlinear terms, F and G, are 
C functions for r > 1 such that 



(4.13) 



i^(0,0)=0, G(0,0) = 

-(o,o) = o, -i^(o,o) = o. 



d{x,X) ' ' d{x,\) 

Since we are locally proving the existence of the stable manifold, we just need the 
local behavior of the dynamics; so first we talk about cut-off functions. The proof 
also requires the discussion on the stability of the nonlinear state dynamics. So the 
following subsection deals the local asymptotic stability of the state dynamics. Then, 
we describe the diagonalization of the bidirectional Hamilton system. Finally, we 
show the existence of A = <(>{x). 

4.1. Cut-ofF Functions. First, we introduce a C°° cut-off function p{ij) : M" — > 
[0, 1] such that 



p{y) 



1, ifO<|?/|<l 
^0, if|j/|>2 

and < p{y) < 1 otherwise. Then we define the functions 



(4.14) 



F(x,A+;e) 
G{x,X+;e) 



F{xp{-),\+p{ — )) 
e e 

G{xp{^),X+p{—)) 
e € 
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forx, A+ G M". Since the F(x,A+) andG(x, A+) agree with F(a;, A+; e) and G'(x, A+; e), 
respectively, for |A+| < e, it suffices to prove the existence of a stable manifold for 
some e > 0. 

Now, we show that there exists A^i > and A^2 > such that 



(4.15) 
(4.16) 
and 
(4.17) 

(4.18) 



\F{x,X;e) - F{S:,X;e)\ < A^ie |a; - i| + |A - A| 
|G(a;,A;e) - G(£,A;e)| < A^ie - i| + |A - A| 



OF 



dix,X) 
dG 



d{x,X) 



(a;, A;e) - 
(.T, A; e) - 



dF 
dix,X) 
dG 



(i,A;e) 
(i,A;e) 



< N2 

< N2 



\x-x\ + \X- X\ 
\x-x\ + \X- X\ 



dix,X) 

Since p{y) and its partial derivatives are continuous functions with compact support 
there exists M > such that 



iy) 



dy 

dy 



< M 



< M 



for all A G R". We also choose M > large enough that 

dF 



(4.19) 
(4.20) 
(4.21) 



dx 
dF 

9A 



(x,A) 



< M\x\ 



{x,X) < M\X\ 



d^F 



ix,X) 



< A/, i,j = l,2 



dx^dX^ 

because of the condition 14.13|l for |a;|, |A| < 1. By the Mean Value Theorem, 

\F{x,X;e) ~ F{i,X;e)\ < \F{x, X; e) - F{i, X;e) + F{S:, X;€) ~ F{S:,X; e)\ 
< \F{x, A; e) - F(£, A; e)| + \F{x, A; e) - F{S:, A; e) 



< 



dF 
dx 



(6,A;e) 



X — x\ 



dF 
9A 



(x,6;e) |A-A| 



where ^1 is between x and x and ^2 is between A and A. Similarly. 

d^F d^F 



_(,,A;.)-— (x,A;.) < 

dF dF 
— (x, A;e) - — (.T, A;e) < 



Next we estimate for < e < i 



dx^ 
d^F 



dXdx 



(a, A) 

(6, A) 



X ~ x\ + 
a: — i| + 



dxdX 
d^F 



5A2 



(2^,6) |A-A| 
(x,6) |A-A|. 



dF 
dx 



(a,6;e) 



< 



ox e e oy e e e 

M|p(^)iiai(|^(^)^ 

e \ dy e e e / 



< M[M+l]e 
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and 



^(^(7)6,7)6) — (-)7+ «3(-)t2 



for <e- 
Let 



It follows that 
(4.22) 



Ni = M^[M + l] 

N2 = M{1 + 2M) + 8M^. 



dF 



(4.23) 



dix,\) 
d^F 



(fi,6;e) 
(6,6;e) 



< N2, i,j = l,2 



dx'dXi 

for l^ili 1^2! < £■ The inequalities above also hold for G as well. 
Thus, 



(4.24) 
(4.25) 
and 
(4.26) 

(4.27) 



|F(x,A;e) -i^(i,A;e)| < A^ie |a; - i| + |A - A| 
|G(a;,A;e) - G(£,A;e)| < NieVx - i\ + \\ - \\ 



dF 



d{x, A) 
dG 



fx, A;e) 



(a;, A; e) 



dF 
d{x,X) 
dG 



(x,A;e) 



(.T, A; e) 



< N2 

< N2 



\x-x\ + \X- A| 
\x-x\ + \\- A| 



9(a;, A) ' ' d{x, A) 
Henceforth we suppress e and write F{x, A), G(x, A) for F{x, A; e), G(x, A; e). 

4.2. Stability of the Nonlinear Dynamics. From Section 2, we mentioned that 
the linear term of the stable manifold for the nonlinear bidirectional Hamiltonian 
dynamics is Px. Then, the local stable manifold is of the form 



(4.28) 



A 



(a;) = Px + ijjix) 



where ipi^) contains all the nonlinear terms. 

Suppose we substitute (|4.28|l into the state dynamics in (|4.12|) . then the nonlinear 
state dynamics becomes 

(/ + BR-^B'P)x+ = Ax - BR-^B'i:{x+) + F{x, Px+ + ijix+)). 

By the Matrix Inversion Lemma ([22), we have that 

(/ + BR-^B'P)-^ = {1 - B{B'PB + R)-^B'P). 

Then, it follows that 



(4.29) 



{A + BK)x + U{x,x+) 



x(0) = xq 
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where K = -{B'PB + R)-^B'P and U{x,x+) = {I + BR-^ B' P)-'^{F{x, Px+ + 
ij{x+)) - BR-^B'ij{x+)). 

The impUcit equation above can be solved. Let T : A/'e(0) C K^" ^ R" such that 

T{x, x+) = x+ - {A + BK)x - f^,{x, x+) =0 

for x,x~^ € JVeiO) where 7Ve(0) is an open neighborhood of radius e around 0. Then, 
for € Afe{0) the Jacobian 

'^-(0) - /-|%(o) 



dx+ dx^ 



= I-{I + BR-^B'P)-^ (^—(0, m)Q^{Q) BR~^B'-^{Q) 
= I. 

because of the condition H4.13|l and only contains nonlinear terms. Then, by 

the Implicit Function Theorem there exists F(.t) such that 

(4.30) x+ = F(a-) 

is equivalent to the earlier state dynamics H4.29(l . Moreover, the linear term of F(a;) 
is {A + BK)x, i.e.; 

(4.31) ¥{x) = (A + BK)x + F^{x) 
because 

= -1\-{A + BK)\ 
= A + BK. 

It follows that Ftp(x) contains only the nonlinear terms and thus, 

(4.32) F^(0) = 
and 

|^(0) = * = !,.. .,n. 

OXi 

The linear part of H4.29|l is 

(4.33) x+ = {A + BK)x. 

Since the eigenvalues of {A + BK) lie strictly inside the unit circle, the term [A + 
BK)^xq — > as fc — > oo. Thus, the system (|4.33|) is asymptotically stable. Also, 
it implies that there exists a unique positive definite P that satisfies the Lyapunov 
equation 

{A + BK)'P{A + BK) - P = -I. 
Now we show the stability of the nonlinear dynamics 

Xk = {A + BK)x + F^{x) 
x{0) = 0. 

We must prove that 

limM^=0; 

x^O \x\ 
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i.e., given any e > and any ip{x) satisfying the conditions 

(4.34) V(0) 

(4.35) \4>{x) - ^p{x)\ < l{e)\x-x\ 

where l{e) — > as e ^ 0, there exists 5 > such that 

— ; — ; — < e whenever \x\ < o. 

\x\ 

We define '0(x) to be the nonhnear term of the stable manifold in (|4.28|) . The con- 
ditions H4.34|l and 1)4. 35(1 will be necessary for the proof of the local stable manifold 
theorem. 
Recall that 

x+ = ¥{x) = {A + BK)x + F^{x). 

Then, 

Q = T{x,x+) = x+ - {A + BK)x - f^,{x,x+) 

= {A + BK)x + F.4,{x) - (A + BK)x - f^,{x, [A + BK)x + F^{x)) 
= F^{x) - {A + BK)x + F^(x)). 

It follows that 

F^{x) = {I + BR-^B'P)-^\F{x,P{{A + BK)x + F^{x))+'il;{{A + BK)x + F^{x))) 

(4.36) -~BR-^B'i){{A + BK)x + F^{x)) . 

Let Bi = + BR-^B'P)~^\l B2 = \\BR-'^B'\l P = ||P|| and a = max, |Ai| where 
\i e a{A + BK) and |Ai| < 1. We have the following from 

\F^{x)\ < MiNie[\x\ + \P{A + BK)x + PF^{x)+i:{{A + BK)x + F^{x)))\] 
+MiM2\yj{iA + BK)x + F^{x)))\. 

because of (|4.24|l . Using the Lipschitz condition l|4.35|l for ^^{x), 

\F^{x)\ < [Bi7Vie + a(Bi7Vie||P||+Bi7Vie;(e)+liB2/(e))]|a;| 
+ [MiNie\\P\\ +BiNiel{e) +RiM2l{e)] \F^{x)\. 

Solving for |i^^(x)|, 

LA^ie + Q;(Bi7Vie|lP|| + BiiVieZ(e) + M2l{e)) 



\F^{x)\ < Y 



iNie + KiNiel{e) +M2lie)) 

-{BiNie+niNiel{e)+B2l{e)) 
+Q(BiJVie|[P||+BiiViei(e)+B2/l 

S>0. Then, 



Let 5 = B^jVieVyj^i?Pr+BSi"^Jw^^^^ For some e > and e > 0, we have that 



Nie + a(MiNie\\P\\+MiNiel{e) + M2l{e)) , , 

- \x\ 



iNie + MiNiel{e) +M2l{e)) 
^ MiNie + a{MiNie\\P\\+MiNiel{€)+M2l{€)) ^ 
- 1 - (BiiVie + BiiVie;(e) + B2?(e)) 

< e. 



Thus, 

F^{x) = o{\x\). 
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Now, we use the Lyapunov argument. Let v{x) = x'Px. Then, 
Av{x) = u(x^) — v{x) 

= x^' Px^ — x' Px 

= [{A + BK)x - F,p{x)]'P[{A + BK)x - F^{x)] - x'Px 
= x'{{A + BK)'P{A + BK) - P)x + 2x'{A + BK)'PF^{x) 
= -\x\'^ + 2x'{A + BK)'PF4.{x). 

since 

\F^{x)\ < ^\x\ 

and 

\2x'{A + BKyPF^{x)\ < 

for some p > 0. Thus, 

Av(x) = --L-L < 0. 
^ ^ 3 

Therefore, the nonlinear dynamics is locally asymptotically stable uniform for all 

gx. 

4.3. Diagonalization of the Hamiltonian Matrix. Recall the bidirectional non- 
linear dynamics in (|4.12() and the condition H4.13|l . 
By substituting 

(4.37) X = Px + ^{x) 
into the state dynamics above H4.12|l . we get 

(4.38) x+ = iA + BK)x + f,p{x,x+) 

where /^(x,x+) = {I + BR-^B'P)-^{F{x,Px+ + il;{x+)) - BR~^B'ij{x+)). 

As we substitute (|4.37|) and H4.38|l into the costate dynamics in (|4.12|) , we also add 
= {BKyX'^ — [BKyX^. Then, the costate dynamics becomes 

\={A + BKy\+ + Qx + g.^,{x, x+). 

where Q ^ Q-K'B' P{A+BK) and g^{x, x+) = G{x, Px++'ip{x+))+K'B'{-ij{x+)~ 
PU{x,x+). 

Thus, the substitution of 

\ = Px + tl;{x) 

into the dynamics (|4.12|) results in a new nonlinear dynamics 



(4.39) 



x+ ' 




' A + BK 




X 




f^{x,x+) 


A 




Q {A + Bxy _ 




A+ 


+ 


_ 9->p{x,x+) _ 



where 

Uix,x+) = {I + BR-^B'P)-\F{x,Tjj{x+)) - BR-^B'iix+)) 

and 

g^ix,x+) = G{x,iPix+))+K'B'i-iix+)~PUix,x+). 
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The nonUnear terms /,/, and 5^ are functions for k > 1 such that 
(4.40) /v, (0,0)^0, 3^,(0,0) = 



(4.41) 



^(0,0) = 0, 



d{x,x+) ' ' d{x,x+) 

because of (|4.13ll . ipi^) only contains nonlinear terms and 

9-0 



(0,0) ==0. 



dx 



(0) = 0. 



Now we introduce the z coordinate by the transformation 

(4.42) X = z + Sx 

for some matrix 5* to block diagonalizc the block lower triangular Hamiltonian matrix 
in (|4.39|) . By substitution, the system H4.39() becomes 

x+ = {A + BK)x + f^{x,x+) 
z ^ iA + BKyz+ + {A + BKyS{A + BK)x-Sx + Qx + h^ix,x+) 

where 

h^ix,x+) = {A + BKySf^{x,x+) + g^{x,x+). 
From the z dynamics above observe that the terms 

{A + BKyS{A + BK)x -Sx + Qx^Q 
where Q = Q - K'B'P{A + BK). Indeed, 

(4.43) - 5 + A'S{A + BK) + K'B'S{A + BK) = -Q + K'B'P{A + BK). 
We know that 

(4.44) -S + A'S{A + BK) = -Q, 

is the discrete-time algebraic Riccati equation (DTARE). Subtracting (|4.44|l from 
(|4.43|l . we have 

K'B'S{A + BK) = K'B'P{A + BK). 

Thus, 

(4.45) S = P 

and S satisfies the DTARE. Therefore, we have a diagonalized system 



(4.46) 
where 

and 



x+ ' 




' A + BK 




X 




U{x,x+) 


z 




{A + BKy 




_ z+ 


+ 


h^{x,x+) 



{I + BR-^B'P)~^{F{x,ij{x+)) - BR-^B'i'{x+)) 



h^{x,x+) = {A + BKyPf^{x,x+) + g^{x,x+). 
The nonlinear terms and h^, are C functions for r > 1 such that 
(4.47) /^(0,0)=0, /i^,(0,0)=0 



(4.48) 



d{x, x+) 



(0,0) =0, 



d{x, x+) 



(0,0)=0. 
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because of H4.40|l and H4.41(l . 

4.4. The Local Stable Manifold Theorem. Given the original dynamics (|4.12() 
we look for the local stable manifold described by A = <p{x). Since the linear term of 
the stable manifold for the system H4.12(l is Px where P is the solution to DTARE 
(j2.9(l , we assume the local stable manifold is of the form 

(4.49) X = Px + i:{x) 

where ip^x) only contains the nonlinear term of (f>{x). In the two-step process of di- 
agonalization of the system (|4.12|l . we introduce the z coordinate through the trans- 
formation 

X — z + Sx. 

Since S = P, it must be that 

z = A — Px = Px + ipi^) ^ Px = ip{x). 

Then, it suffices to prove existence of the local stable manifold z = ip{x) for the 
diagonalized system (14.46(1 . In order to show the existence of 2; = ip{x), we use the 
Contraction Mapping Principle (CMP). To invoke the CMP, we will need a map 
T : X — > X that is a contraction on a complete metric space X. 

Theorem 4.1. Given the dynamics in 14-4 
that are C functions satisfying the conditions 
point € M^", there exists a local stable manifold 

(4.50) z = i:{x) 
around the fixed point where ip is a function. 
Proof: 

First notice that and are cut-off functions. It follows that h^, is also a cut-off 
function. It suffices to prove the theorem for some e > since the cut-off functions 
f^{x,x^;e) and h^,{x,x'^;e) agree with f^{x,x'^) and h^{x,x~^) for jxl, |a;"''| < e. 

By (0231)- and ifjTTjl - lfi:^ . there exists Ni, N2 > such 
that 

\U{x,y;e) - fi,{x,y;e)\ < 
\h^{x,y;e) - h^{x,y;e)\ < 



with the nonlinear terms f^ and 

and {4-4-^ o,''^^ o, hyperbolic fixed 



(4.51) 
(4.52) 
and 
(4.53) 

(4.54) 



iVie 



dfi, 



d{x,y) 
dh^ 



{x,y;e) 
{x,y;e) 



d{x,y) 
dh^ 



(i,y;e) 
(i,y;e) 



d{x,y) ' ' d{x,y) 
Moreover, from the bound (|4.22|) we know the following 



x\ + \y- y\ 



x\ + \y- y\ 



(4.55) 
and 
(4.56) 

foriV>Oand |aU6l < £■ 



d[x,y) 



(a,6;e) 



d{x,y) 



(ei,e2;e) 



< Ne 



< Ne. 
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Henceforth we suppress e and write /^(x, y), h^(x, y) for f^{x, y; e), h^(x, y; e). 
Let /(e) with /(O) = and ■(/; e X C C"(|a;| < e) where X is space of V' : R" — > K' 
such that 

(4.57) ^(0) = 

(4.58) \^{x) - ^1j{x)\ < l{e)\x~x\ 

for X, X ^ ■6c (0) C M" where Bc(0) is cfosed bah around with radius e. We define 

(4.59) W^PW ^ sup 

a:|<e 

To show X is a complete metric space it suffices to show that X is closed since 
X C C''({|a;| < e}). We take a sequence {ipn} S X such that — > V' in C° norm. 
For large N > n, |i/'„(a;) - ^ix)\ < f for aU x G 8^(0). Then, 

\->p{x) - 'lp{x)\ < llpix) - 'lpn{x)\ + \lp7iix) - 1pnix)\ + \lp7iix) - 'lp{x)\ 

< i+l(e)\x-x\ + ^. 

By letting e — > 0, we have that \ip{x) — ipix)\ < /(e) |a; — x\. Thus, tp is a Lipschitz 
function. Similarly, the condition H4.57(l is easily satisfied. It follows that 

mO)-0\ < 1^/^(0) - 7/^„(0)| + |V'„(0) - 0| <e. 

Thus, V' G X. Hence X is closed. Moreover, X is a complete metric space with the 
norm defined on (|4.59() . 

Solving the z dynamics in H4.46|l via the variation of constants formula, we have 

k-l 

(4.60) z, = {A' + K'B')''-^Zk + J2i^' + K'B'y-^h4xi,xi+i) 

1=] 

for j < k. Let j = and fc = oo, then H4.6()(l changes to 

oo 

(4.61) zo = ^(A' + X'B')'/i^(x,,x,+i) 

1=0 

We define a mapping T : X — > X by 

oo 

(4.62) {Ti:){xo) = Y.{A' + K'B'yh.4xi,xi+i). 

1=0 

where xi,xi+i satisfy 

x+ = A + BKx + f.4,{x,x+). 

From this fixed point equation, we look for the solution 

{Tip)(xo) = iixo). 

We must show Tip G X and prove T is a contraction on X. 

Suppose xo = is the initial condition. Clearly from the equation H4.30|l - (|4.32|l . 
Xfe = for all k. Together with Xk = for all k and the condition H4.48|l ^.^^(O, 0) = 0, 
wc have 

(4.63) rV'(O) = 0. 

Hence Tip satisfies the condition (|4.57() . 

We now prove the Lipschitz condition H4.58|l for Tip. 



X 
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For -0 e X and the initial conditions xq, xq G K", we denote Xk = a;(fc, xo, "0) to 
be the solution of the state dynamics. 

x+ = {A + BK)x + f^,{x,x+) 

X{0) = Xq. 

Similarly, for G X and the initial conditions xq € R", let Xk = x(k,xo,il)) the 
solution of 

x+ = {A + BK)x + f^,{x,x+) 

X{0) = Xq. 

Recall a = maxj \Xj \ where Xj G a{A + BK) and \Xj\ < 1. Using the estimate (|4.51|l . 
at one-time step 

\xk+i ~ Xk+i \ < a\xk -Xk\+ Ni [\xk -Xk\ + \xk+i - Xk+i\] ■ 
For 1 - iVie > 0, 

\Xk+l - Xk+l \ < ^FA; - Xk\ 



l~Nie' 



and recursively, 



As long as 



then 



l^^fe - Xk\ < 



a + Nie\ 
1-Nie) 

1 - a 



\Xo - Xo\ 



e < 



2Ni ' 
a + NiE 



1-Nie 



< 1. 



Thus, 
(4.64) 



\Xk - Xk\ < \xo - Xo\. 



Using the bounds and 



|r0(xo)-T0(xo)| < 



^(A' + K'B'y{h^{xi,xi+i) - h^{xi,xi+i)) 



1=0 



< y^g' \h^{xi,xi+i) - h^{xi,xi+i) 



t=o 



< 



^a'Nie[\xi - xi\ + \xi+i - xi+i \ 

1=0 



1=0 

2Nie 



< 



Let 



1 — a 



\xo - Xo\ 



2Nie 
1 -a' 
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Notice that 1(e) ^ as e ^ 0. Thus, 

\T'ip{xo) ~ Tip{xo)\ < l{e)\xo - xo\ 

and so (Tip) satisfies the condition H4.58|l for e > suSicicntly smah. 
Hence T maps from X ^ X. 
Next we show T is a contraction on X. 
We express the solutions to the state dynamics 

Xk = x(fc,xo,i/') 

and 

Xk = x{k,xo,tp) 

in the imphcit form, 

fe-i 

Xk = {A + BKfx^ + ^(A + BKf-^-^ f^xj.xj+i) 

3=0 

and 

fc-i 

Xk = {A + BKfx^ + ^(A + BKf-^-^ f^{x„x,+i), 
j=o 

respectively. 

We now denote Xj = x{j,xo,ip) and Xj = x(j,xo,ip) be the solutions to the state 
dynamics and satisfy the implicit form equations 

fc-i 

Xk ^{A + BKfxa + ^(A + BKf-^~\U{x,,x,+i) 

and 

fc-i 

Xk = {A + BKfxQ + ^(A + BKf-^-^ff,{x,,Xj+i), 
j=o 

respectively. 

The estimates H4.51|l - (|4.52|) with the trajectories x{j, xq, ip) and x{j, xq, ip) becomes 

(4.65) \U{x,y) - f4,{x,y)\ < ri{e)\y - y\ + r2{e)\\ip ~ + r3{e)\x - x\ 

(4.66) \h^{x,y) - h^{x,y)\ < ri{e)\y - y\ + r2{e)\\ip - -ipW + r3{e)\x - x\ 
where 

ri(e) = nislie) + ni^2<^ + ni^3l{e)e 
r2(e) = n24e + ri2,2e^ 
''3(e) = nse 

and Hi J are positive constants. Observe that ri{e) — > as e — > 0. 
At one-time step, 

\xk+i-Xk+i\ < m2ie)\xk - Xk\ + m3{e)\\ip - ipW 

where 

a + r3(e) 



l-ri(e) 

and 

l-ri(e)- 



TO2(e) = 
TO3(e) 
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By invoking Gronwall's inequality 1)3. l|l and assuming that for some small e > 

TO2(e) < 1, 

then 

fe-i 



(4.67) 



< 



1 - m2(e) 
"^3(e) 



1 - m2(e) 

With the bounds and ifTHTjl . we get 



IV'-V'II 



\T4,{xo)-T^{xa)\ < 



J2i^' + K'B'y{h,p{xi,xi+i) - h^{xi,xi+i)) 

l=Q 

oc 

1=0 
oc 

y^ct' [ri(e)|x;+i ~ xi+i\ + r2(e)||?/> - +r3(e)|xi - xi\ 

"^3(e) II , 



< 



< 



1=0 

oc 

E' 

(=0 



2(ri(e) + r3(e)) 



1 - TO2(e) 



■^2(e)||V'-V'll 



It follows that 



where 



T^{xo)-T^{xo)\ < c(e)||^-^|| 
2m3(e)(ri(e) +r3(e)) , r2(e) 



c(e) 



(1 -m2(e))(l -a) 1-a 
Notice that c(e) — *■ as e — > 0. Thus, for sufficiently small e > 

c(e) < 1. 

Hence, T is a contraction on X for e sufficiently small. Therefore, there exists a unique 
■0 € X such that 

?/' = r?/;. 

Let a;, z satisfy 

a;+ = A + BKx + f^{x,x+) 

and 

z+ = + BKYz + g^{x, 
respectively. Choose z = ^/'(a;). Using the deffiiition of the contraction T in H4.62|l . 
we have 

zo = {T^)ixo) 



J2iA' + K'B'yh^ixi,xi+,). 



1=0 



It follows that (x, z) is a solution to the difference equation (|4.46() and (|4.5Q|I defines 
a invariant manifold. Thus, (|4.11|) is the graph of a invariant manifold. As 
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described in one can show straight forwardly that H4.11|l defines C" invariant 
manifold given the data are C smooth. I 

5. Some Properties 
5.1. Eigenstructure. Recall from Section 1 the bidirectional linear Hamiltonian dy- 



(5.68) 
where 

Definition 5.1. Suppose 
(5.69) 



" x+ ' 




X 


A 


= H 


A+ 



A 
Q 

Sx 
fiSX 



-BR-^B' 
A' 



fiSx 
5X 



Then we call fi the eigenvalue of the dynamics f5.68\) . 

We would like to show that the linear bidirectional Hamiltonian matrix H is hy- 
perbolic; i.e., the eigenvalue of the dynamics (|5.68|l lies strictly inside and outside the 
unit circle. 

Theorem 5.2. If ii is an eigenvalue of the dynamics 1^5. 6 S\) satisfying the relation 
\5.69\) . then ^ is also an eigenvalue of the dynamics 1^5. Ot^) . 

Proof: First, we decompose H into 

/ 1 r Q 

/ A 



Let's call 



Q 

A 



A' 

-BR-^B' 



A' 

-BR-^B' 

" / 
^lI 



Ai/ 
/ 



and notice that § is symmetric. 
We rewrite l)5.6H|l to 



It follows that 

(5.70) 

We denote 



5x 
SX 



Sx 
6X 



Sx 
SX 

Sx 
SX 



Observe that 1 is an eigenvalue of from H5.70|l . Then, 

det [I - H,,] = 0=^det[l - H^] 

where 
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Again, 1 is an eigenvalue of H' ; i.e. 



Sx 
SX 



dx 
6X 



By multiplying 
Thus, 



to equation above, we have 



/ 
/ 



H 

6X 



Sx 



^5x 



dx 



/ 
/ 



^SX 

n 

5x 



Hence, - is an eigenvalue of H5.68|l . 



Note that the dynamics 
singularity of A. 



admits the infinite eigenvalues, and oo, due to the 



5.2. Symplectic Form. The nonlinear dynamics tangent to 1)4.12(1 as derived explic- 
itly using perturbation technique in citeNa02 is 



(5.71) 



" 5x+ ' 




SX 





H 



X+x 



5x 
SX+ 



where H^+x, Hxx, and Hx+x+ are the partial derivatives of the Hamiltonian defined in 
and {Sx, SX) are tangent vectors in T^x.\)M for M = {{x, X)\x G M", A e M"}. 
The nondcgenerate and bilinear symplectic two-form ^l : Ti^x,\)-M x T^x,\)J^ i-^ K is 

r2(w, w) — v' Jw 

with 

ft{v, w) = —il{w, v) 

where the symplectic matrix, 

/ 

-/ 



J 



and 





Sx 




Sx 


V = 


SX 


, w = 


Tx 



We would like to show that under the tangent dynamics H5.71|l . the two- form J7 is 
invariant; i.e., 

(5.72) ^^(w,^) = 0(t;+,'u;+). 
Then, 

r2(w,w) = v' Jw 

= -sy^Tx 4- Sx'^sx. 

By substituting the tangent dynamics of Sx in H5.71|l into the above equation, we have 
^{v,w) = -(F3;.T'5a; + i^A+i;'5A+)'fa + (5a-'(iJ^^(5a; + ifA+x'5A+) 

(5.73) = -SX"^H'^^Jx^SxRx^x^"^- 
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Similarly, 

(5.74) = -5\+H'y^+ji + 5xHx+a^5)^- 
Since ()5.74|l and (|5.73|l arc equal, then 

Thus, for any two tangent vectors satisfying the dynamics H5.71|l . the value of Vi does 
not change. 

6. Lagrangian Submanifold 
The two-form calculated from the last section is 

^l{v,w) = -5X^' Hy^+,^5x + 5x' Hx+x5>^^ ■ 

We calculate the state dynamics tangent to (|4.46|l around the trajectories (xj,a;j+i) 
as 

(6.75) faj+i = ((A + B/i) + ^(xJ,XJ+l))faJ■ 

{x3,Xj + i)SXj + i 

By the Inverse Function Theorem, we can choose e > small enough so that 
,Xk+i) is invertible for \xk\, \xk+i \ < e because of (|4.4UII and hence, 

^ ^^'^ (0,0)=/. 



dxk+i 

It follows that (|6.75() is equivalent to 

Sxk+i = (l - f'^"^' ixk,Xk+i)) ({A + BK) + ^:^{xk,Xk+i)]Sxk 



dxk+i ' / V dx 



and 



(6.76) 5xk+i = TT f / - ^'^'^ {xi,x,+i] 

dxi+i 



1=0 



iA + BK) + ^{x,,x,+i)yxo. 
for \xk\, \xk+i\ < e. As we let k oo, we have Xk ^ 0, (0) ^ 0. It follows 



that from 

Sxk+i ^{A + BKfSxQ -» 0. 

Thus, 

n{v, w) = ~6\+' Hx+x^x + 6x'Hx+JX^ 

since dxk — > as fc — > cxd for w, w restricted to the tangent dynamics. Hence, the local 
stable manifold is a Lagrangian submanifold. 
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Denote Ws as the local stable manifold described by the graph A = The 
basis for T(^x,4>{x))Ws is of the form 

Xi 



d 

dxi 



where i = 1, . 



Then, the two-form 
where 



dxi 



Xi 



d 

dxj 



Xi 



G T{x,<t,{x))Wa 



is equivalent to 



d(l)i d(l)j 



for i, j ~ 1, 



dxj dxi 

The equation above implies that (l){x) is closed. Then, by Stokes' Theorem there 
exists TT e r{W^) such that 



ix) = T^fx) where (/)(0) = 
ox 



(6.77) 

locally on some neighborhood of 0. Thus, there exists tt e C such that 
(6.78) 



Hence, the local stable manifold A in (|6.78|l is the gradient of the optimal cost for 
the bidirectional Hamiltonian dynamics H2.3|l . 

7. Application to Dynamic Programming Equations 

Recall from Section 2 the formulation of a discrete in time infinite horizon optimal 
control problem of minimizing the cost functional, 



■n ^ — ^ 



k = 



subject to the dynamics 



x{0) = .To 
We assume l{x,u) is convex in x and u so that 



g s 

S* R 



> 
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and let i? > 0. In addition, it is assumed that the pair {A, B) is stabihzable and the 
pair {A,Q^/'^) is detectable. 

Given that a:(0) = the optimal value function t:{x) is defined by 

oo 

tt[xo) = minS^ l{xk,Uk). 

u ^ — ^ 

fc=0 

This value function satisfies a functional equation, called the dynamic programming 
equation. The optimal feedback k{x) is constructed from the dynamic programming 
equation. We state the optimality principle: 

Theorem 7.1. Discrete-Time Optimality Principle: 
(7.79) 7r(x) = min{7r(/(x, u)) + l{x, u)} 

U 

Proof: We have that 

7r(xo) = m:u\(S^ l{xk,Uk)} 

ua ^ — ' 

fc=0 



min{/(xo,uo) + ^(xfc, Mfe)} 



k=l 

= min{/(xo, uq) + 7r(a;i)} 

Generalizing the optimality principle at the fc*''-step, we have 
(7.80) 7r(x) = min{7r(a;+) + l{x, u)}. 

u 

■ 

The optimality equation (|7.79|l is the first equation of the dynamic programming 
equations. An optimal policy u* ~ k{x) must satisfy 

tt{x) - 7r{f{x, u*)) ~ l{x, u*) = 

if we assume convexity of the LHS of (|7.80(l . We can find u* through 

d{Tr{x) — Tr{f{x, u)) — l{x, u) 



du 

which by the chain rule becomes 







|^(/(a^,u))|^(a:^,'«) + ^(^''") = ^ 

Thus, t:{x) and k,{x) satisfy these equations, the Dynamic Programming Equations 
(DPE): 

(7.81) tt{x) -T:{f{x,u)) -l{x,u)) = 

(7.82) |(/(.,.))|(.,.) + £(.,.) . 

To show the existence of the local solutions, ■k{x) and n{x), we use the Pontryagin 
Maximum Principle (Theorem 2.1). From the condition (|2.6|l and the consequence of 
Local Stable Manifold Theorem where A is function of x; i.e., A = ^{x), we have 

u* = k(x) ~ argmin H(x, ——(a;), v). 

V ox 
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The above equation is equivalent to 

dH diT 

which is essentially (|7^ . Thus, |f (a;) and k{x) solve (|7^ . Since 



A 



Si 



from the PMP and 



A 

9x ' 



it follows that 

Integrating H7.83|l w.r.t. a;, we get 

Tr(x) — 7r(/(x, k(x))) — /(x, = 

which is lEHJ. Therefore, tt and k solve the DPE (jTHTUT^ . Furthermore tt e 
and K G since I G C and / G C"^^ in the Hamiltonian. 
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